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CREATING AN INCENTIVE TO AUTHOR USEFUL ITEM REVIEWS 

TECHNICAL FIELD 

The present invention is directed to the fields of electronic commerce and 
on-line communities. 

BACKGROUND 

Web merchants sell items such as products, services, and data via the World 
Wide Web ("the Web"). Because most items sold by a web merchant result in a profit, 
measured by the amount by which the item's price exceeds its cost, web merchants have a 
strong incentive to increase the rate at which they sell items. 

It is common for web merchants to design their web sites to include content 
that helps to draw interest to the web sites and to particular items m order to increase the 
number of customers that may buy these items. As one example of such content, some 
web merchants include item reviews on their web sites, which typically provide 
additional information about an item and, in the case of a positive review, an endorsement 
of the item. For example, some web merchants furnish book reviews on their web sites. 

In some cases, web merchants pay professional reviewers to prepare item 
reviews. In these cases, the often-significant cost of procurmg professional reviews can 
have a negative impact on the web merchant's profitability. Additionally, some readers of 
professional reviews may be distrustfid of the evaluation of a professional reviewer. 

In other cases, volunteers, such as customers, are solicited to prepare item 
reviews. While volunteer reviews can be procured much less expensively than 
professional reviews and may have more appeal to readers that prefer reviews from those 
they perceive to more substantially share their perspective, volunteer review programs 
often have significant disadvantages of their own. 
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For example, it can often be difficult to convince volunteers to prepare item 
reviews. Of those volunteer reviews that are prepared and submitted, a significant 
portion may be of little use to prospective purchasers for a variety of reasons; such 
reviews may be poorly written, fail to adequately support their conclusions, contain 
5 subject matter irrelevant to the item, engage in ad hominem attacks on the creator of the 
reviewed item, etc. 

In view of the above-discussed disadvantages of conventional types of 
reviews, a more effective approach to obtaining useful item reviews from volunteers 
would have significant utility. 

10 BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1 is a high-level block diagram showing a typical environment in 
which the facility operates. 

Figure 2 is a display diagram showing a typical display presented by the 
facility showing a list of top-ranked reviewers. 
15 Figure 3 is a display diagram showing a typical display presented by the 

facility containing more detailed information about a particular top-ranked reviewer. 

Figure 4 is a display diagram showing a typical display presented by the 
facility containing detailed information about an item. 

Figure 5 is a display diagram showing a typical display presented by the 
20 facility incorporating information about two randomly-selected top-ranked reviewers. 

Figure 6 is a display diagram showing a typical display presented by the 
facility to enable a user to submit a new review. 

Figure 7 is a display diagram showing a typical display presented by the 
facility containing guidelines for preparing an item review. 
25 Figure 8 is a display diagram showing a typical display presented by the 

facility containing sample reviews. 

Figure 9 is a flow diagram showing steps typically performed by the facility 
in order to process a new item review submitted by a reviewer. 

Figure 10 is a flow diagram showing steps typically performed by the 
30 facility in order to process votes received for published item reviews. 
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Figure 11 is a flow diagram showing steps typically performed by the 
facility to update reviewer scores and rankings. 

DETAILED DESCRIPTION 

A software facility for creating an incentive to author useful item reviews 
("the facility") is described. In some embodiments, the facility is used by a web merchant 
to obtain content that, when incorporated in the web merchanf s web site, helps to draw 
interest to the web site and to increase the number of customers that may buy items from 
the web merchant. The facility provides such an incentive by rewarding the authors of 
reviews found to be useful by their readers, such as by prominently displaying their 
names and ranks as authors of useful reviews prominently on the web merchant's web 
site. 

In some cases, review authors (also called "reviewers") submit reviews each 
relating to a specific item sold by the web merchant. The web merchant displays these 
reviews to customers in conjunction with the items to which they relate, together with a 
control — such as buttons — ^fhat may be used by viewing customers to vote on how useful 
the review is to them. For example, in some embodiments, the control enables viewing 
customers to specify either that they found the review useful or that they did not find the 
review useful The facility stores these votes, and periodically uses them to score each 
reviewer in terms of the number of reviews that the reviewer submitted and their level of 
usefulness. 

Each time new scores are generated, the facility ranks the reviewers in 
descending order of their scores. These rank values are then typically displayed liberally 
in conjunction with information about the top-ranked reviewers. For example, the facility 
may display a list of the top-ranked reviewers, in which the rank of each reviewer is 
indicated. The facility may also display a shorter list of reviewers randomly selected 
from among the top-ranked reviewers, such as from the top 100 reviewers. Such displays 
may include such information as the names of these reviewers, their pictures, and 
biographical sketches. The facility may display in conjunction with any occurrence of a 
top-ranked reviewer's name an indication of that reviewer's rank. For example, 
occurrences of the name of the reviewer having rank 1 may be accompanied by a "#1 
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Reviewer" message, which the name of the reviewer having rank 89 may be accompanied 
by a "Top 100 Reviewer" message. Such displays may occur on pages such as a home 
page for the reviewer, or in conjunction with the reviewer's reviews. 

By promoting reviewers that submit reviews that prove to be useful to their 
readers, the facility motivates many of the users visiting the web merchant's web site to 
prepare and submit useful item reviews. This in tum typically increases the level of 
traffic to the web merchant's web site (in terms of number of visitors, average frequency 
with which a single user visits, and average duration of a visit), the number of sales 
completed and therefore the profitability of the web merchant, and the level of 
satisfaction of item purchasers, both with respect to their purchases specifically and with 
respect to the web merchant generally. 

Figure 1 is a high-level block diagram showing a typical environment in 
which the facility operates. The block diagram shows several client computer systems, 
such as cUent computer systems 110, 120, and 130. Each of the cHent computer systems 
has a web client computer program for browsing the World Wide Web, such as web 
clients 111, 121, and 131. The cHent computer systems are connected via the Memet 
140 to a server computer system 150 hosting the facility. Those skilled in the art will 
recognize that client computer systems could be connected to the server computer system 
by networks other than the Intemet, however. 

The server computer system 150 contains a memory 160. The memory 160 
preferably contains web merchant software 161 incorporating both the facility 162 and 
data typically used by facility, such as item reviews 163, votes 164 on item reviews, and 
item order and/or sales data 165. The memory preferably further contains a web server 
computer program 166 for delivering web pages in response to requests from web clients. 
While items 161-166 are preferably stored in memory while being used, those skilled in 
the art will appreciate that these items, or portions of them, maybe be transferred between 
memory and a persistent storage device 172 for purposes of memory management and 
data integrity. The server computer system further contains one or more central 
processing units (CPU) 171 for executmg programs, such as programs 161, 162, and 166, 
and a computer-readable medium drive 173 for reading information or installing 
programs such as the facility from computer-readable media, such as a floppy disk, a CD- 
ROM, or a DVD. 
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While various embodiments are described in terms in the environment 
described above, those skilled in the art will appreciate that the facility may be 
implemented m a variety of other environments including a single, monolithic computer 
system, as well as various other combinations of computer systems or similar devices 
cormected in various ways. 

Figures 2-5 show examples of ways in which the facility promotes, and 
thereby rewards, top-ranked reviewers. Figure 2 is a display diagram showing a typical 
display presented by the facility showing a Ust of top-ranked reviewers. The display 200 
is displayed as part of the web site of web merchant Amazon.com. The display includes 
laudatory language 201 congratulating the top-ranked reviewers. The display also 
includes an ordered list 210 of the top-ranked reviewers. List 210 is comprised of 
ordered entries, each corresponding to one top-ranked reviewer, such as entries 220, 230, 
and 240, As an example, entry 220 contains information about the highest-ranked 
reviewer, Harriet Klausner. This entry contains the reviewer's rank 221, as well as a 
graphical characterization 222 of the rank. The entry further contains the name of the 
reviewer 223, which is a link that the user can select in order to view more detailed 
information about this reviewer. The entry further contains an indication 224 of the total 
number of reviews authored by this reviewer. The entry also contains further information 
225 about the reviewer, typically provided by the reviewer him or herself. This 
information includes a link 226 that may be selected by the user to display additional 
information about the reviewer. Some entries contain an image of the reviewer, such as 
image 237 shown in entry 230. 

In addition to the list 210 of detailed entries about the top-ranked reviewers, 
the display also contains a more abbreviated list 250 of the top-ranked reviewers. In this 
list, each entry is merely the rank value and the name of the reviewer, which is a link that 
may be selected by the user to display additional information about the reviewer. 

Figure 3 is a display diagram showing a typical display presented by the 
facility containing more detailed information about a particular top-ranked reviewer. In 
particular, the display 300 shown in Figure 3 is displayed by the facility when the user 
selects link 243 for the reviewer Frank Behrens shown in Figure 2. The display includes 
a profile 310 for the reviewer, which include such information as the reviewer's name 
311, a nickname 312 for the reviewer, an email address 313 for the reviewer, the current 

[24976-8042/SL003738.550] 5 4/24/01 



rank 314 of the reviewer, the beginning of a biographical sketch 315 of the reviewer, a 
link 316 to the entire biographical sketch of the reviewer, a count 317 of the number of 
reviews submitted by this reviewer, a count 318 of the number of positive votes cast for 
the reviews of this reviewer, and a graphical indication 319 of the rank of the reviewer. 
5 The display further contains each of the reviews submitted by the reviewer. As an 
example, the display contains information 320 about the reviewer's most recent review. 
This information includes information 321 about the reviewed item, such as the title, 
artist, format, price, and availability of the item, as well as a link 322 tiiat may be used to 
display more information about the item, and a control 323 that may be used to place an 

10 order for the item. The information 320 also includes a grade 324 assigned by the 
reviewer to the item as part of the review — ^here the grade shown is four stars out of five 
stars; the review's title 325; the date 326 on which the review was submitted; and the text 
327 of the review. The display also contains a section 330 containing links to item 
categories for which the reviewer has submitted reviews. 

15 Figure 4 is a display diagram showing a typical display presented by the 

facility containing detailed information about an item. In this case, tiie item is the 
Tchaikosvky Symphony No. 6 DVD whose review is shown in Figure 3. The display 400 
shown in Figure 4 is displayed in response to the selection of link 322 shown in Figure 3. 
In addition to other information about the item, the display includes a list of reviews 

20 submitted for the item. These include review 413 submitted by the reviewer Frank 
Behrens. The review 410 includes the reviewer's name 411, which is a link to the display 
shown in Figure 3; a graphical indication 412 for the current rank of the reviewer; the 
item grade 413 assigned to the item as part of the review; the review title 414; the text of 
the review; and voting buttons 415 and 416. A user that is undecided about buying this 

25 item may read review 410. Based upon various aspects of the review, the user may either 
find the review helpful in deciding whether to purchase the item or unhelpful in deciding 
whether to purchase the item. Such a user may select button 415 if the review was 
helpful, or button 416 if the review was not helpful. In some embodiments, such votes 
are the basis on which the facility scores and ranks this reviewer. The display further 

30 includes a link 421 that the user may select to submit his or her own review of the 
Tchaikosvky Symphony No. 6 DVD item. 



[24976-8042/SL003738 550] 



6 



4/24/01 



Figure 5 is a display diagram showing a typical display presented by the 
facility incorporating information about two randomly-selected top-ranked reviewers. 
The display 500 includes a list 510 of randomly-selected top-ranking reviewers. The list 
is comprised of one or more entries, such as entries 520 and 530 shown here, which each 
relate to one randomly-selected top-ranked reviewer. For example, entry 520 relates to 
reviewer number 14, and includes the reviewer's rank value 521; the reviewer's name 522, 
which is a link which that the user may select in order to display more information about 
this reviewer; the number of reviews submitted by this reviewer 523; and the number of 
positive votes 524 received by this reviewer's reviews. Some entries contain an image of 
the corresponding reviewer, such as image 535 in entry 530. The display also includes a 
link 540 to the top reviewer's list shown in Figure 2. 

Figures 6-8 show an example of an approach to obtaining an item review 
submitted by a reviewer. Figure 6 is a display diagram showing a typical display 
presented by the facility to enable a user to submit a new review. The user typically 
selects this display in conjunction with a particular item, for which the user can submit a 
review. For example, display 600 is displayed in response to the user's selection of link 
421 in the display containing detailed information about the Tchaikovsky Symphony No. 
6 DVD shown in Figure 4. 

The display includes the identity of the user, which is attributed as the 
identity of the reviewer. To attribute a different identity to the reviewer, the user can 
select link 602. The display includes information 603 identifying the item to be 
reviewed. The display includes a control 604 used by the user to specify a grade or rating 
for the reviewed item. The display includes field 605 for entering a title for the review. 
The display includes field 606 for entering the text of the review. The user may select 
radio button 607 in order to display a reviewer name with the review, which may edited 
in field 608. Alternatively, the user may select radio button 609 to make the review 
anonymous and prevent the review from being associated with the reviewer's name. The 
display also includes field 610 for entering the author's location. Before preparing the 
review as described, the user may select link 621 in order to display guidelines for 
preparing the review. The user may also select link 622 in order to display one or more 
example customer reviews demonstrating compUance with the review guidelines. When 
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the user has assembled the review to be submitted using the above described aspects of 
the display, the user selects button 611. 

Figure 7 is a display diagram shov^ing a typical display presented by the 
facility containing guidelines for preparing an item review. The display 700 contains 
typical guidelines for this activity. 

Figure 8 is a display diagram showing a typical display presented by the 
facility containing sample reviews. The display 800 includes sample reviews, such as 
sample review 810. In some embodiments, sample reviews are chosen from among the 
reviews of top-ranked reviewers, such as from the reviews of the top-ranked 100 
reviewers. 

Figure 9 is a flow diagram showing steps typically performed by the facility 
in order to process a new item review submitted by a reviewer. In step 901, the facility 
receives a new item review from a reviewer. Such a new item review may be received in 
a variety of ways, including receiving it using a web-based form as shown in Figure 6, 
receiving it via electronic mail, receiving it in a docimient uploaded to an FTP server, 
receiving it in paper form, receiving it in audio form, etc. 

In step 902, the facility assigns the review identifier to the review that will 
be used to identify the review and stores the received review in conjunction with the 
assigned review identifier. In step 903, the facility creates an association between the 
assigned review identifier and an identifier for the reviewer. As an example, Table 1 
below shows information stored in a data structure, and, in particular, rows stored in a 
table, in order to create such an association. 

review identifier reviewer identifier 

96335807 449603151134 

96335808 230651087912 

96335809 449603151135 

Table 1 
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For example, the first row shown in Table 1 indicates that the item review having review 
identifier 96335807 was submitted by the reviewer having reviewer identifier 
449603151134. 

In step 904, the facility obtains editorial approval for the review. As an 
example, the facility may in step 904 add the review or its review identifier to a queue of 
new reviews to be subjected to a manual editorial review process. Step 904 is optional, 
and may be applied selectively, such as to reviews submitted by reviewers that have a 
reviewer score below a certain threshold, or a reviewer rank value above a certain 
threshold. Alternatively, the new review may be added to such a queue at different points 
depending upon the score or rank of the submitting reviewer. 

In step 905, after editorial approval has been received for the review, the 
facility makes the review available on the web site. This may involve, for example, 
incorporating the review in a display such as those shown in Figures 3, 4, and 8. Again, 
step 905 may be performed in a manner sensitive to the score and/or rank of the 
submitting reviewer. Afl:er step 905, these steps typically conclude. 

Figure 10 is a flow diagram showing steps typically performed by the 
facility in order to process votes received for published item reviews. In step 1001, the 
facility receives a vote on a review from a voter. For example, the facility may receive an 
indication that a user pressed a button Hke button 415 shown in Figure 4 indicating that a 
review was helpful to the user, or a button like button 416 shown in Figure 4 indicating 
that a review was not helpful to the user. Such a vote typically identifies a review to 
which the vote relates, an identifier identifying the voter, the date and time at which the 
vote was cast, and the value of the vote, i.e., whether the voter indicated that the review 
was helpful or not helpful. In step 1002, the facility stores this information together. For 
example, this information may be stored in an entry of a data structure, such as a row of a 
table, as shown below in Table 2. 

review identifier voter identifier date/time helpful 

96335807 919654002315 04/11/2001 16:04:02 

43227116 310956681822 04/11/2001 16:04:02 

84325510 919654002315 04/11/2001 16:04:03 



yes 
yes 
no 
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Table 2 



For example, it can be seen from the first row shown in Table 2 that a vote on the review 
having review identifier 96335807, cast by a voter having voter identifier 919654002315, 
was cast at 16:04:02 on 04/11/2001, and that this vote indicated that the review was 
helpfiiL After step 1002, these steps typically conclude. 

Figure 11 is a flow diagram showing steps typically performed by the 
facility to update reviewer scores and rankuigs. The facility typically performs these 
steps periodically, such as once per day. In step 1101, the facility identifies reviewers 
whose reviews have received at least one new vote. In some embodiments, step 1101 
involves selecting from the stored votes those votes cast on the date immediately 
preceding the current date. The review identifiers in the selected votes are used to 
identify the reviewer identifiers corresponding to the selected votes. 

In steps 1102-1104, the facility loops through each reviewer identified in 
step 1101 as having received at least one new vote. In step 1103, the facility calculates 
an updated reviewer score for the current reviewer. The facility is preferably 
configurable to use a wide variety of different approaches to determining reviewer scores. 
In one approach, the facility calculates a reviewer score based upon scores calculated for 
each review of the reviewer. In accordance with this approach, for each review, the 
facility applies formula (1) below. 

Score^eview - 2(A + B + C)-(D + E) (1) 

where A is the number of reviews submitted by the author for which at least 3 positive 
indications were received, B is the number of reviews submitted by the author for which 
at least 10 positive indications were received, C is the number of reviews submitted by 
the author that were the first reviews submitted for the corresponding item and for which 
at least 3 positive indications were received, D is the number of reviews submitted by the 
author for which at least 3 more negative indications than positive indications were 
received, and E is the number of reviews submitted by the author for which at least 10 
more negative indications than positive indications were received. 

In some embodiments, the facility excludes from values A through E any 
votes that (1) were subsequent to the first vote cast by the same voter on the same review, 
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or (2) were subsequent to the fifth vote by the same voter on any review by the same 
reviewer. Various other approaches to calculating a score for a review may also be 
employed by the facility, such as those in which fewer or more factors are considered, or 
such factors are weighted differently. Additional factors that various embodiments may 
consider includes such factors as whether the review is the first review submitted for a 
particular item, and the number of orders that have been placed for an item by a customer 
to which the review has been displayed. 

The score for a reviewer is generated by summing the scores of the 
reviewer's reviews as shown below in formula (2). 

ScOrCreviewer ~ ^ ScOrCreview (2) 

reviewer's 
reviews 

In various embodiments, a reviewer's score may be determined by combining the scores 
for the reviewer's reviews in a variety of other ways. 

After step 1103, the faciUty continues in step 1104. hi step 1104, if 
additional identified reviewers remain for processing, the facihty continues in step 1102 
to process the next identified reviewer, else the facihty continues in step 1105. In step 
1105, the facility attributes a rank value to each reviewer based upon their reviewers 
scores, as updated. In step 1 106, the facility proceeds to utilize the reviewer rank values 
attributed in step 1105 in any web pages of the web's web site that reference these rank 
values. This may involve, for example, utilizing the attributed reviewer rank values in a 
display such as those shown in Figures 2-5. After step 1106, these steps typically 
conclude. 

It will be understood by those skilled in the art that the above-described 
facility could be adapted or extended in various ways. For example, the facility may be 
operated by others besides web merchants. A variety of different types of rewards may 
be offered to reviewers submitting useful reviews. The usefulness of reviews may be 
discerned in a variety of different ways. While the foregoing description makes reference 
to preferred embodiments, the scope of the invention is defmed solely by the claims that 
follow and the elements recited therein. 
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